Feature Extraction Apparatus, Feature Extraction Method, and Feature Extraction Program

ABSTRACT

To extract a feature advantageous for classification and correlation by using the information difficult to be acquired even when it is impossible to acquire the information difficult-to-be-acquired from all individuals. Sub-information input device inputs information difficult to be acquired and accumulates the inputted sub-information. Main information input device inputs information easy to be acquired as main information, and accumulates the inputted main information. Sub-information selection device evaluates a category attribution degree of each sub-information accumulated and selects the sub-information of a high category attribution degree. The correlation feature extraction device uses the sub-information selected by the sub-information selection device as the feature extraction filter, and extracts a feature corresponding to the main information from a correlation between the main information and the sub-information.

TECHNICAL FIELD/BACKGROUND OF THE INVENTION

The present invention relates to a feature extraction apparatus, afeature extraction method, and a feature extraction program forextracting features for classification and correlation of data.

BACKGROUND ART

In a data group including plural individual pieces, two or more types ofinformation such as image and three-dimensional information aresometimes used for performing classification and correlation withrespect to each individual piece of data. The accuracy of classificationand correlation can be enhanced generally by using plural types ofinformation.

However, the difficulty level in acquiring information (easiness toacquire information) differs depending on the type of the information.For instance, in an example of image and three-dimensional information,the image can be easily acquired by a camera. However, a range finder orthe like needs to be used to acquire the three-dimensional information.The three-dimensional information is more difficult to acquire than theimage since the range finder is generally not widespread used as withcameras. Also, it is expensive and its use conditions are limited.

If the difficulty level in acquiring information differs as describedabove, the information amount of the type of information with highdifficulty level in acquiring, that is, difficult to be acquired,becomes less than the information of the type of information with lowdifficulty level in acquiring, that is, easy to be acquired. In suchcase, the data lacking in information of the type of high acquiringdifficulty is removed from the classification and classification target,or, correlation and classification are performed by using only theinformation of the type of low acquiring difficulty. With this, however,the accuracy of classification and correlation becomes difficult to beenhanced.

Patent document 1 discloses an image classification device (defectclassification device) for automatically classifying a defected image.Patent document 2 discloses a correlation extraction device of an imagefeature quantity. Patent document 3 discloses a face feature extractiondevice.

Patent document 1: Japanese Laid-Open Patent Publication No. 2001-188906(paragraph 0051)

Patent document 2: Japanese Laid-Open Patent Publication No. 2003-157439(paragraph 0022)

Patent document 3: Japanese Laid-Open Patent Publication No. 2004-21924(paragraphs 0028-0044)

DISCLOSURE OF THE INVENTION

“Information that is not necessarily acquirable from all individuals” isreferred to as information difficult-to-be-acquired. It is preferablethat a feature advantageous for classification and correlation can beextracted by using the information difficult to be acquired that hasbeen able to be acquired, even when it is impossible to acquire theinformation difficult-to-be-acquired from all individuals. Here, featurerefers to a feature which can be used in classification and correlationof data.

Further, feature extraction suited for classification and correlation ispreferably performed.

Furthermore, feature extraction dependent on an internal structure ofthe data such as appearance is preferably performed.

It is an object of the present invention to provide a feature extractionapparatus, a feature extraction method, and a feature extraction programcapable of extracting a feature advantageous for classification andcorrelation by using information difficult to be acquired even when itis impossible to acquire the information difficult-to-be-acquired fromall individuals. Another object is to provide a feature extractionapparatus, a feature extraction method, and a feature extraction programcapable of performing feature extraction suited for classification andcorrelation. Still another object is to provide a feature extractionapparatus, a feature extraction method, and a feature extraction programcapable of performing feature extraction dependent on an internalstructure of the data such as appearance.

A feature extraction apparatus according to the present inventionrelates to a feature extraction apparatus for extracting, from maininformation obtained from an individual piece of data, a feature of theindividual piece; the feature extraction device including a correlationfeature extraction device for calculating a feature quantity of anindividual piece, based on a correlation between the main informationand sub-information which is different from the main information, andusing the main information and the sub-information.

The feature extraction device may include a sub-information storagedevice (e.g., sub-information input device 20) for storing thesub-information classified into categories in advance, and asub-information selection device for calculating a category attributiondegree of each sub-information and selecting sub-information whoseattribution degree is larger than a predetermined reference, wherein thecorrelation feature extraction device calculates the feature quantity ofthe individual piece based on the correlation between the maininformation and the sub-information, by using the main information andthe sub-information which is selected by the sub-information selectiondevice.

The sub-information selection device may perform a main componentanalysis of the sub-information, and calculate the attribution degreeusing a reconfiguration error in a category to which the sub-informationbeing a target for calculating the attribution degree belongs and anaverage of the reconfiguration error in categories other than therelevant category.

The correlation feature extraction device may calculate the featurequantity using a difference between the main information and thesub-information having the same dimension.

A feature extraction method according to the present invention relatesto a feature extraction method for extracting, from main informationobtained from an individual piece of data, a feature of the individualpiece. In this method, a correlation feature extraction devicecalculates a feature quantity of an individual piece based on acorrelation between the main information and sub-information which isdifferent from the main information, by using the main information andthe sub-information.

In this method, the sub-information storage device may store thesub-information classified into categories in advance, thesub-information selection device may calculate a category attributiondegree of each sub-information and select sub-information whoseattribution degree is larger than a predetermined reference; and thecorrelation feature extraction device may calculate the feature quantityof the individual piece based on the correlation between the maininformation and the sub-information, by using the main information andthe sub-information which is selected by the sub-information selectiondevice.

The sub-information selection device may perform a main componentanalysis of the sub-information, and calculate the attribution degree byusing a reconfiguration error in a category to which the sub-informationbeing a target for calculating the attribution degree belongs and anaverage of the reconfiguration error in categories other than therelevant category.

The correlation feature extraction device may calculate the featurequantity using a difference between the main information and thesub-information having the same dimension.

A feature extraction program according to the present invention relatesto a feature extraction program loaded on a computer for extracting,from main information obtained from an individual piece of data, afeature of the individual piece, the computer including asub-information storage device (e.g., storage apparatus 703) for storingsub-information which is different from the main information and beingclassified into categories in advance; the program causing the computerto execute a correlation feature extraction process of calculating afeature quantity of an individual based on a correlation between themain information and the sub-information, by using the main informationand the sub-information.

The program may further cause the computer to execute a sub-informationselection process of calculating a category attribution degree of eachsub-information and selecting sub-information whose attribution degreeis larger than a predetermined reference, wherein the feature quantityof the individual piece is calculated based on the correlation betweenthe main information and the sub-information, by using the maininformation and the selected sub-information in the correlation featureextraction process.

The program may further cause the computer to perform a main componentanalysis of the sub-information, and calculate the attribution degreeusing a reconfiguration error in a category to which the sub-informationbeing a target for calculating the attribution degree belongs and anaverage of the reconfiguration error in categories other than therelevant category in the sub-information feature selection process.

The program may further cause the computer to calculate the featurequantity using a difference between the main information and thesub-information having the same dimension in the correlation featureextraction.

In the present invention, the correlation feature extraction devicecalculates the feature quantity of the individual piece based on thecorrelation between the main information and the sub-information whichis different from the main information, by using the main informationand the sub-information. Therefore, a feature dependent on the internalstructure of the data such as appearance can be extracted.

The sub-information selection device calculates the category attributiondegree of each sub-information and selects the sub-information whoseattribution degree is larger than a predetermined reference, and thecorrelation feature extraction device calculates the feature quantity ofthe individual piece using the main information and the selectedsub-information. Therefore, the feature can be extracted using thesub-information even when the sub-information cannot be acquired fromall the individuals.

Further, since the sub-information classified into categories in advanceis used, the feature quantity having information which is unique witheach category can be calculated. As a result, the feature quantitysuited for classification of the data can be calculated.

Furthermore, with arranging the sub-information storage device forstoring the sub-information classified into categories in advance, thefeature quantity can be calculated when the main information is input.Therefore, the design of the system for performing data classificationcan be facilitated.

According to the present invention, a feature quantity advantageous forclassification and correlation can be calculated by using informationdifficult to be acquired even when it is impossible to acquire theinformation difficult-to-be-acquired from all individuals. Also, afeature quantity dependent on the internal configuration of the datasuch as appearance can be calculated. Further, the feature quantitysuited for classification of the data can be calculated. Furthermore,the design of the system for performing data classification can befacilitated.

BEST MODE FOR CARRYING OUT THE INVENTION

In the present invention, the information of a representative individualpiece which is difficult to be acquired is selected as the information(difficult-to-be-acquired information) that is not necessarilyacquirable from all individuals. Such difficult-to-be-acquiredinformation is then used as a feature extraction filter for extracting afeature from the information easy to be acquired. The information whichis easy to be acquired is information that can be acquired from eachindividual piece. Individual piece is one of the data in each category.

The representative individual piece of the informationdifficult-to-be-acquired is selected by using a category attributiondegree of each of information difficult-to-be-acquired. “High categoryattribution degree” refers to a state in which an approximate expressionby other members of the category to which the piece of data belongs iseasy, and an approximate expression by the member of other categories isdifficult.

The attribution degree is calculated in the following manner (moredetailed description of the attribution degree calculation will bedescribed later). With regards to the each individual piece of data, theapproximate expression by a member other than itself in the category towhich the relevant individual data belongs is calculated, and theexpression error (error in self-category) in that case is recorded. Theapproximate expression by all the members of each category to which theindividual piece being focused does not belong (approximate expressionwith a same method as the approximate expression by the members in thesame category) is calculated, and the expression error (error innon-belonging category) in that case is calculated. Furthermore, anaverage of the errors in all the non-belonging categories is calculated.The attribution degree is represented as a difference between theaverage of the errors in all the non-belonging categories and the errorin the self-category, or as a ratio between the average of the errors inall the non-belonging categories and the error in the self-category.

Information with high attribution degree, as defined above, is being theinformation in which the feature of the self-category with respect toother categories is emphasized. The performance of data classificationcan be enhanced by using information with high attribution degree.

In the case of correlation, the self-category is considered as one data,while the other categories are considered as one. The definition ofattribution degree is expressed as the reconfiguration error in thecategories which are configured other than the self-category.

The individual piece with high attribution degree out of the informationdifficult to be acquired (more specifically, difficult-to-be-acquiredinformation acquired from the relevant individual piece) is selected andassumed as a feature extraction filter. The feature extraction isperformed on the information easy-to-be-acquired by using a correlationbetween the information easy-to-be-acquired and the informationdifficult-to-be-acquired. As the simplest example, a difference betweenthe information easy-to-be-acquired and the informationdifficult-to-be-acquired is used as the correlation between theinformation easy-to-be-acquired and the informationdifficult-to-be-acquired. If the relationship is clear such as withimage and illumination space base, such relationship can be used forcorrelation. Thus, the correlation reflects the internal structure ofthe data, and the feature to be extracted is also influenced by thedata.

The difficult-to-be-acquired information used as the feature extractionfilter is selected based on the attribution degree rather than beingarbitrarily selected. In other words, information(difficult-to-be-acquired information) advantageous for classificationand correlation representing the property of the category is selected.Therefore, a feature advantageous for classification and correlation canbe extracted.

The reconfiguration error and the error minimum estimate error will bedescribed below. First, the reconfiguration error will be described. Thetrue information (e.g., true image) is assumed as It. The information Itis expressed as It=f(P1, P2, . . . , Pn) by elements P1, . . . , Pn anda function f(·). The information reconfigured by insufficient elementsP1, . . . , Pn and the function f(·) is assumed as Ia. The informationIa is expressed as Ia=f(P1, . . . , Pn) by the insufficient elements P1,. . . , Pn. In Ia, the true information It and the information Ia do notmatch since the elements are insufficient. The difference between them(It and Ia) is referred to as reconfiguration error.

When estimating the information It by f(P1, . . . , Pn), estimation ismade through an estimation method (e.g., least squares estimation) suchthat the reconfiguration error becomes a minimum. The error in this casebecomes the error minimum estimate error.

The exemplary embodiments of the invention will now be described withreference to the drawings.

FIRST EXEMPLARY EMBODIMENT

FIG. 1 is a block diagram showing a configuration example of a firstexemplary embodiment of the present invention. A feature extractionapparatus according to the first exemplary embodiment includes a maininformation input device 10, a sub-information input device 20, asub-information selection device 30, and a correlation featureextraction device 40.

The main information input device 10 inputs information easy to beacquired as main information, and accumulates the inputted maininformation. The sub-information input device 20 inputs informationdifficult to be acquired as sub-information, and accumulates theinputted sub-information.

The sub-information selection device 30 selects the informationdifficult-to-be-acquired that is effective in classification andcorrelation of data from the sub-information inputted and accumulated inthe sub-information input device 20.

The correlation feature extraction device 40 extracts a feature relatedto the correlation between the main information and the sub-informationfrom the sub-information which is selected by the sub-informationselection device 30.

The extracted feature is referenced when carrying out classification andcorrelation of the data. The feature extraction apparatus may include alearning/identification device 50 for performing classification andcorrelation of the data. However, the learning/identification device 50is not an essential component of the feature extraction apparatus. Ifthe feature extraction apparatus includes the learning/identificationdevice 50, the learning/identification device 50 becomes the outputdestination to which the correlation feature extraction device 40outputs the feature.

The operation will now be described.

FIG. 2 is a flowchart showing an example of progress of the process bythe feature extraction apparatus in the first exemplary embodiment. Thesub-information input device 20 inputs information difficult to beacquired as sub-information, and accumulates (stores) the inputtedsub-information (step S1). The accumulation of the sub-information instep S1 is performed prior to step S2. The sub-information input device20 collectively inputs each sub-information, and accumulates the same.An example of information difficult to be acquired that becomes thesub-information includes information acquired by a special device suchas a range finder and a thermo-camera. More specifically, theinformation includes three-dimensional information acquired by the rangefinder, the thermo image acquired by the thermo-camera, and the like.The sub-information is information that is more difficult to be acquiredcompared to the main information.

The number of pieces of sub-information inputted and accumulated in stepS1 does not need to be as large as the number of pieces of maininformation inputted and accumulated in step S2 described below. Thatis, the number of pieces of sub-information may be less than the numberof pieces of main information. Furthermore, the sub-information inputtedand accumulated in step S1 does not necessarily need to be theinformation acquired from the same individual piece as the maininformation inputted and accumulated in step S2.

Each sub-information inputted and accumulated in step S1 is classifiedinto categories in advance.

The main information input device 10 inputs information easy to beacquired as main information, and accumulates the inputted maininformation (step S2). An example of information easy to be acquiredthat becomes the main information includes an image acquired by a devicein widespread use such as a camera. The image is information frequentlyused in classification and correlation of data. An input andaccumulation mode of the main information of the main information inputdevice 10 may be a mode of collectively inputting and accumulating eachmain information. Furthermore, a mode of sequentially inputting andsequentially accumulating each main information, and performing theprocesses after step S3 may be adopted. Further, the informationgenerally used in the system of performing classification andcorrelation of data may be used to be inputted to the main informationinput device 10 as the main information.

The sub-information selection device 30 evaluates the categoryattribution degree of each sub-information accumulated in thesub-information input device 20, and selects the sub-information withhigh category attribution degree (step S3). The sub-informationselection device 30 performs selection of the sub-information bycategory attribution degree in the following manner, for example.

The sub-information selection device 30 selects the sub-informationbeing the target for calculating the attribution degree, converts therelevant sub-information to a column vector, and checks the category(referred to as i) to which the relevant sub-information belongs. Here,the column vectorized sub-information is expressed as A. Thesub-information selection device 30 performs a main component analysiswith members other than the sub-information A of the sub-information ofthe category I (Leave-One-Out). The result of the main componentanalysis is P^(i)=[P1 ^(i), . . . , Pk^(i), . . . Px^(i)]. Thesub-information selection device 30 calculates t^(i) shown below. Inequation (1), “T” represents transposition.

t ^(i) =[t0^(i) , . . . , tx ^(i)]^(T) =A ^(T) P ^(i)  Equation (1)

The sub-information selection device 30 then calculates areconfiguration error g^(self) in the self-category (category to whichthe sub-information selected as the target for calculating theattribution degree belongs). The sub-information selection device 30calculates the reconfiguration error g^(self) with the followingequation (2).

g ^(self)=norm (A−P ^(i) t ^(i))=norm(A−P ^(i) A ^(T) P ^(i))  Equation(2)

Here, norm (·) is the norm amount such as least square norm, Manhattandistance, and the like.

Suppose there are N−1 counter-categories of category 1, . . . j, . . .N(j≠i) to which the sub-information A does not belong. Thesub-information selection device 30 performs the main component analysiswith each N−1 categories. The result of the main component analysis ineach category is P^(j)=[P1 ^(j), . . . , Pk^(j), . . . Px^(j)].

The sub-information selection device 30 then calculates areconfiguration error average (average of the reconfiguration errors)g^(others) of the counter-categories. The sub-information selectiondevice 30 then calculates the reconfiguration error average g^(others)with the following equation (3).

$\begin{matrix}{g^{others} = {\frac{1}{N - 1}{\sum\limits_{j \neq 1}^{N}{{norm}\left( {A - {P^{j}A^{T}P^{j}}} \right)}}}} & {{Equation}\mspace{14mu} (3)}\end{matrix}$

The category attribution degree (referred to as g^(belong)) of thesub-information is defined as the difference between the reconfigurationerror average g^(others) and the reconfiguration error g^(self) in theself-category as shown in the following equation (4), or the ratio ofthe reconfiguration error average g^(others) and the reconfigurationerror g^(self) in the self-category (ratio of the reconfiguration erroraverage g^(others) with respect to the reconfiguration error g^(self))as shown in the following equation (5).

g ^(belong) =g ^(others) −g ^(self)  Equation (4)

g ^(belong) =g ^(others) /g ^(self)  Equation (5)

Therefore, the sub-information selection device 30 calculates thecategory attribution degree g^(belong) of the sub-information accordingto equation (4) and equation (5). The sub-information selection device30 evaluates the category attribution degree of each sub-information andselects the sub-information with high category attribution degree in theabove manner.

The preferred sub-information g^(belong) (i.e., sub-informationg^(belong) to be selected) is the sub-information of the case when theerror of the self-category (reconfiguration error in the self-category)g^(self) is small, and the error of the counter-category(reconfiguration error average of the counter-category) g^(others) islarge. If the error g^(self) of the self-category is smaller and theerror g^(others) of the opposing category is larger, the attributiondegree g^(belong) becomes a larger value in the definition ofattribution degree in both equation (4) and equation (5).

If the attribution degree g^(belong) is defined as in equation (4), theproperties of the self-category with respect to other categories are notprovided when g^(belong) is smaller than or equal to 0. Accordingly, ifthe attribution degree g^(belong) is defined as in equation (4), thesub-information selection device 30 preferably excludes thesub-information whose attribution degree g^(belong) is smaller than orequal to 0 from the selection target. In other words, thesub-information selection device 30 preferably selects thesub-information whose attribution degree g^(belong) is greater than 0.

If the attribution degree g^(belong) is defined as shown in equation(5), the properties of the self-category with respect to othercategories are not provided if g^(belong) is smaller than or equal to 1.Accordingly, if the attribution degree g^(belong) is defined as inequation (5), the sub-information selection device 30 preferablyexcludes the sub-information whose attribution degree g^(belong) issmaller than or equal to 1 from the selection target. In other words,the sub-information selection device 30 preferably selects thesub-information whose attribution degree g^(belong) is greater than 1.

If the number of sub-information that can be used as a filter is limitedby restriction of resources in the correlation feature extraction device40, the sub-information selection device 30 selects the sub-informationin the descending order according to large attribution degree g^(belong)value.

When performing correlation using the extracted feature, onesub-information to be the target for calculating the attribution degreeconfigures one category with only the sub-information itself.Accordingly, g^(self)=0 when defining the attribution degree g^(belong)as shown in equation (4). Furthermore, g^(self)=1 when defining theattribution degree g^(belong) as shown in equation (5).

After step S3, the correlation feature extraction device 40 uses thesub-information selected by the sub-information selection device 30 asthe feature extraction filter, and extracts a feature corresponding tothe main information from a correlation between the main information andthe sub-information (step S4).

Suppose there exists a function f that satisfies I_(i)=f(X_(i)) betweenthe main information (referred to as I_(i)) and the sub-information(referred to as X_(i)) of the same individual piece. In this case, theerror minimum estimate error F₀(I^(i)) expressed as equation (6) usingthe sub-information X0 of another individual piece can be also expectedto have the effect as the feature quantity representing the property ofthe main information I^(i).

F ₀(I _(i))=norm(I _(i) −f(X ₀))  Equation (6)

The error minimum estimate error F₀(I_(i)) obtained by using thesub-information X₀ selected based on the category attribution degree inequation (6) can be used as a feature quantity reflecting the categoryattribution degree. Therefore, the correlation feature extraction device40 calculates F₀(I_(i)) by equation (6) using the sub-information X₀selected in step S3 and the main information accumulated in the maininformation input device 10, where the calculation result F₀(I_(i)) isassumed as the feature. The correlation feature extraction device 40outputs the calculated feature (feature quantity).

The feature quantity output by the correlation feature extraction device40 is input to the learning/identification system (system forclassifying and correlating the data. Learning/identification device 50in the example shown in FIG. 1), where the function f does notnecessarily need to be a correct function if the learning/identificationsystem corrects the feature quantity. In this case, the function f maybe a simple mapping 1. If a known function representing the relationshipbetween the main information and the sub-information exists, such knownfunction is preferably used. An example of a case of using the knownfunction includes a case where the main information Ii is the image andthe sub-information Xi is the illumination space base. The illuminationspace base is the information amount reflecting a three-dimensionalimage and the reflectivity of the face, and, according to a document“Peter N. Belhumeur, David J Kriegman, What Is the Set of Images of anObject Under All Possible Illumination Conditions?”, InternationalJournal of Computer Vision, Vol. no. 28, 245-260, 998”, a matrix of N×3is obtained assuming the image I_(i) is a N-dimensional vertical vector,and the relationship of the following equation (7) is met.

I_(i)=X_(is)  Equation (7)

In equation (7), s is referred to as an illumination vector, and is athree-dimensional vector representing the direction and the magnitude ofthe illumination. In this case, when the sub-information X₀ of anotherindividual piece (sub-information acquired from an individual piecewhich is different from the main information) is used, the correlationfeature extraction device 40 calculates the s (illumination vector) atwhich the value of equation (8) shown below becomes a minimum, and usesF₀(I_(i)) in such case as the feature.

F ₀(I _(i))=norm(I _(i) −X _(0s))  Equation (8)

A case where the main information I_(i) is an image of a living being,and the sub-information X_(i) is a thermo-image of the relevant livingbeing is considered as an example of when the function representing therelationship between the main information and the sub-information isunknown. In this case, the definite relationship between the maininformation and the sub-information is not known. However, both (maininformation and sub-information) are images and have the same dimension,and thus the function f can be assumed as simple mapping 1. In thiscase, the correlation feature extraction device 40 may calculate thefeature F0(Ii) with the following equation (9).

F ₀(I _(i))=norm(I _(i) −X ₀)  Equation (9)

That is, the correlation feature extraction device 40 calculates thefeature quantity using the difference between the main information I_(i)and the sub-information X₀ as shown in equation (9) when the dimensionsof the main information and the sub-information are the same. Thefeature quantity F₀(I_(i)) obtained through equation (9) is not afeature quantity having a direct basis as in the feature quantityF₀(I_(i)) obtained through equation (8). However, it is a featurequantity reflecting the category attribution degree by the maininformation and the sub-information, and the property that the categoryattribution degree by the main information and the sub-information isreflected can be used in the learning/identification system.

The correlation feature extraction device 40 outputs the obtainedfeature quantity to the learning/identification device 50, and thelearning/identification device 50 performs classification andcorrelation of data based on the feature quantity. As described above,however, the learning/identification device 50 is not an essentialcomponent of the present invention.

According to the present invention, the sub-information selection device30 selects the sub-information with high category attribution degreefrom the sub-information (information difficult-to-be-acquired), and thecorrelation feature extraction device 40 uses such sub-information toextract a feature. Therefore, a feature advantageous for classificationand correlation can be extracted by using information difficult to beacquired even when it is impossible to acquire the informationdifficult-to-be-acquired from all individuals.

The correlation feature extraction device 40 uses the sub-informationclassified into categories in advance as a feature extraction filter,and extracts (calculates) the feature corresponding to the maininformation from the correlation of the main information and thesub-information. As a result, a feature which is unique with eachcategory can be extracted. This feature is advantageous forclassification as it reflects the feature of the category to beclassified. That is, feature extraction suited for classification can becarried out in the present invention. For instance, assumesub-information αs, βs, γs, and main information αm, βm, γm are obtainedwith respect to categories α, β, γ, respectively. If the feature isextracted using the sub-information αs, βs, γs with respect to the maininformation αm, feature vectors reflecting the category, such as, strongcorrelation between αm and αs, weak correlation between αm and βs, andweak correlation between αm and γs are obtained. Similarly, if thefeature is extracted using the sub-information αs, βs, γs with respectto the main information βm, feature vectors reflecting the category,such as, weak correlation between βm and αs, strong correlation betweenβm and αs, and weak correlation between βm and γs are obtained.Moreover, if the feature is extracted using the sub-information αs, βs,γs with respect to the main information γm, feature vectors reflectingthe category, such as, weak correlation between γm and αs, weakcorrelation between γm and βs, and strong correlation between γm and γsare obtained.

In the first exemplary embodiment, the correlation feature extractiondevice 40 uses the sub-information (informationdifficult-to-be-acquired) as the feature extraction filter, and extractsa feature from the main information (information easy-to-be-acquired)using the feature extraction filter. Thus, a feature dependent on theinternal structure of the data such as appearance can be extracted.

The sub-information selection device 30 selects effectivesub-information (sub-information in which the value of attributiondegree is larger than the reference), and the correlation featureextraction device 40 uses the sub-information as the feature extractionfilter. The sub-information to be selected is input to thesub-information input device 20 in advance and accumulated therein.Therefore, the feature can be extracted by inputting the maininformation to the main information input device 10. As a result, thedesign of a system for performing classification and correlation can befacilitated.

The flowchart shown in FIG. 2 is illustrative, and the process does notnecessarily need to be performed in the order shown in FIG. 2. Forinstance, after inputting and accumulating the sub-information inadvance, the sub-information selection device 30 may calculate thecategory attribution degree for each sub-information and select thesub-information based on the magnitude of the attribution degree. Themain information input device 10 may thereafter sequentially input andaccumulate the main information, and the correlation feature extractiondevice 40 may extract a feature corresponding to the main informationusing the relevant main information and the selected sub-information. Inother words, step S3 may be executed before step S2.

SECOND EXEMPLARY EMBODIMENT

FIG. 3 is a block diagram showing a configuration example of a secondexemplary embodiment of the invention. A feature extraction apparatus ofthe second exemplary embodiment includes each device corresponding toeach apparatus shown in the first exemplary embodiment. That is, thefeature extraction apparatus of the second exemplary embodiment includesa main information input apparatus 100, a sub-information inputapparatus 200, a sub-information selection apparatus 300, and acorrelation feature extraction apparatus 400, as shown in FIG. 3.Further, learning/identification apparatus 500 may be arranged.

The main information input apparatus 100 corresponds to the maininformation input device 10 in the first exemplary embodiment, andoperates similar to the main information input device 10. Thesub-information input apparatus 200 corresponds to the sub-informationinput device 20 in the first exemplary embodiment, and operates similarto the sub-information input device 20. The sub-information selectionapparatus 300 corresponds to the sub-information selection device 30 inthe first exemplary embodiment, and operates similar to thesub-information selection device 30. The correlation feature extractionapparatus 400 corresponds to the correlation feature extraction device40 in the first exemplary embodiment, and operates similar to thecorrelation feature extraction device 40. The learning/identificationapparatus 500 corresponds to the learning/identification device 50 inthe first exemplary embodiment, and operates similar to thelearning/identification device 50. The operation of each apparatus 100to 500 is similar to that of each device 10 to 50 (see FIG. 1) in thefirst exemplary embodiment, and the feature extraction apparatus of thesecond exemplary embodiment performs operations similar to steps S1 toS4 (see FIG. 2). Thus, detailed description related to the operationwill be omitted. The configuration of each apparatus will be describedbelow.

FIG. 4 is an explanatory view showing a configuration example of themain information input apparatus 100. The main information inputapparatus 100 is realized by a storage apparatus 101 such as a disc or amemory. The mode of input and accumulation of the main information inthe main information input apparatus 100 may be a mode of collectivelyinputting and accumulating (storing) each main information. Furthermore,a mode of sequentially inputting and sequentially accumulating each maininformation, and performing the processes after step S3 may be adopted.In the mode of accumulating great amount of main information in acollective process, a high-capacity magnetic disc etc. is preferablyused for the storage apparatus 101. In the mode of sequentiallyinputting and accumulating the main information and performing theprocesses after step S3, high-speed access DRAM etc. is preferably usedfor the storage apparatus 101. The main information stored in thestorage apparatus 101 is read by the correlation feature extractionapparatus 400 in step S4.

FIG. 5 is an explanatory view showing a configuration example of thesub-information input apparatus 200. The sub-information input apparatus200 is realized by a storage apparatus 201 such as a disc or a memory.Assumption is made that the sub-information input apparatus 200collectively inputs and accumulates the sub-information. Therefore, ahigh-capacity magnetic disc etc. is preferably used for the storageapparatus 201. The sub-information stored in the storage apparatus 201is read by the sub-information selection apparatus 300 in step S3.

FIG. 6 is an explanatory view showing a configuration example of thesub-information selection apparatus 300. The sub-information selectionapparatus 300 includes a calculation apparatus 301 such as a CPU and astorage apparatus 302 such as a disc or a memory. The calculationapparatus 301 reads each sub-information from the storage apparatus 201of the sub-information input apparatus 200, calculates the attributiondegree, and selects the sub-information which attribution degree islarger than a value that becomes a reference in step S3 (see FIG. 2).This process is the same as the operation of the sub-informationselection device 30 in the first exemplary embodiment. The calculationapparatus 301 accumulates the selected sub-information in the storageapparatus 302. The sub-information accumulated in the storage apparatus302 is read by the correlation feature extraction apparatus 400 in stepS4.

FIG. 7 is an explanatory view showing a configuration example of thecorrelation feature extraction apparatus 400. The correlation featureextraction apparatus 400 includes a calculation apparatus 401 such as aCPU. The calculation apparatus 401 reads the main information from thestorage apparatus 101 of the main information input apparatus 100, andreads the selected sub-information from the storage apparatus 302 of thesub-information selection apparatus 300 in step S4 (see FIG. 2). Thecalculation apparatus 401 then calculates a feature corresponding to themain information from the correlation of the main information and thesub-information by equation (6). The calculation apparatus 401 outputsthe calculated feature. If the feature extraction apparatus includes thelearning/identification apparatus 500, the learning/identificationapparatus 500 becomes the output destination of the feature.

FIG. 8 is an explanatory view showing a configuration example of thelearning/identification apparatus 500. The learning/identificationapparatus 500 includes a calculation apparatus 501 such as a CPU, astorage apparatus 502 such as a DRAM memory for accumulating thetemporary calculation result enabling high-speed access, and ahigh-capacity storage apparatus 503 such as a disc for storing thecalculation result of the calculation apparatus 501. The calculationapparatus 501 uses the feature output from the correlation featureextraction apparatus 400 to perform the calculation related tolearning/identification, and accumulates the calculation result in thehigh-capacity storage apparatus 503.

Effects similar to the first exemplary embodiment are obtained in thesecond exemplary embodiment.

As described in the first exemplary embodiment, the process does notnecessarily need to be proceeded in the order shown in FIG. 2. Forinstance, after inputting and accumulating the sub-information inadvance, the sub-information selection apparatus 300 may calculate thecategory attribution degree for each sub-information and select thesub-information based on the magnitude of the attribution degree. Themain information input apparatus 100 may thereafter sequentially inputand accumulate the main information, and the correlation featureextraction apparatus 400 may extract a feature corresponding to the maininformation using the relevant main information and the selectedsub-information. In other words, step S3 may be executed before step S2.

THIRD EXEMPLARY EMBODIMENT

FIG. 9 is a block diagram showing a configuration example of a thirdexemplary embodiment of the invention. A feature extraction apparatus ofthe third exemplary embodiment includes a calculation apparatus 701, aprogram storage unit 702, a storage apparatus 703, a main informationinput unit 704, a sub-information input unit 705, and an output unit706.

The calculation apparatus 701 controls the entire feature extractionapparatus 700 according to a feature extraction program stored in theprogram storage unit 702, and executes the process of each stepdescribed in the first exemplary embodiment. The calculation apparatus701 is realized by a CPU and the like. The feature extraction apparatus700 includes a buffer (not shown) for loading the main information andthe sub-information used by the calculation apparatus 701 in thecalculation.

The program storage unit 702 stores the feature extraction program forcausing the calculation apparatus 701 to execute the process.

The storage apparatus 703 stores the inputted main information andsub-information.

The main information input unit 704 is an input interface of the maininformation, and the sub-information input unit 705 is an inputinterface of the sub-information. The input interface of the maininformation and the input interface of the sub-information may becommon. The output unit 706 is an interface for outputting the featurecalculated by the calculation apparatus 701.

The calculation apparatus 701 inputs the main information through themain information input unit 704, and accumulates the main information inthe storage apparatus 703. Similarly, the calculation apparatus 701inputs the sub-information through the sub-information input unit 705,and accumulates the sub-information in the storage apparatus 703.

The calculation apparatus 701 selects the sub-information, similar tothe sub-information selection device 30 in the first exemplaryembodiment, and extracts a feature, similar to the correlation featureextraction device 40 in the first exemplary embodiment. The calculationapparatus 701 outputs the extracted feature through the output unit 706.

An input mode of the information includes a mode (referred to as firstinput mode) in which a data group (sub-information is lacking in someindividual piece of data) including the main information and thesub-information is collectively input, and a mode (referred to as secondinput mode) in which a data group including the main information and adata group including the sub-information are independent, a small numberof sub-information is input to and accumulated in the feature extractionapparatus in advance, and the main information is sequentially inputafterward.

FIG. 10 is a flowchart showing an example of progress of the process ofthe calculation apparatus 701 in the first input mode. The maininformation and the sub-information of the data group (sub-informationof some individual piece of data is lacking, and number ofsub-information is less than number of sub-information) are assumed tobe already collectively input and accumulated in the storage apparatus703.

The calculation apparatus 701 loads one of the main information of eachindividual piece to the buffer (step S100). The calculation apparatus701 determines whether or not the sub-information of the individualpiece same as the individual piece from which the main information isacquired can be loaded (steps S101, S102). If the sub-information of theindividual piece same as the individual piece from which the maininformation is acquired does not exist and cannot be loaded (N of stepS102), the process returns to step S100, and the processes after stepS100 are repeated.

If the sub-information of the individual piece same as the individualpiece from which the main information is acquired exists and can beloaded (Y of step S102), the calculation apparatus 701 loads thesub-information to the buffer (step S103).

After step S103, the calculation apparatus 701 determines whether or notall the main information are loaded to the buffer (step S104). If notall the main information are loaded (N of step S104), the processreturns to step S100, and the processes after step S100 are repeated. Ifall the main information are loaded (Y of step S104), the calculationapparatus 701 calculates the attribution degree of each sub-information(step S105). The calculation apparatus 701 selects the sub-informationaccording to the magnitude of the attribution degree (step S106). Thecalculation of the attribution degree and the selection of thesub-information in steps S105, S106 may be performed similar to thecalculation and the selection described in the first exemplaryembodiment.

Thereafter, the calculation apparatus 701 extracts (calculates) afeature of the main information with the selected sub-information as thefeature extraction filter (step S107). This calculation may be performedsimilar to the calculation described in the first exemplary embodiment.

After step S107, the calculation apparatus 701 determines whether or notthe feature is calculated for all the main information (step S108). Ifthe feature is not calculated for all the main information (N of stepS108), the process returns to step S107, and the processes after stepS107 are repeated. The calculation apparatus 701 outputs the calculatedfeature.

FIG. 11 is a flowchart showing an example of progress of the process ofthe calculation apparatus 701 in the second input mode. In the secondinput mode, the main information and the sub-information do notnecessarily need to be acquired from the same individual piece, andconsideration is made in acquiring only the sub-information in advance.Here, assumption is made that such sub-information is input in advanceand accumulated in the storage apparatus 703.

The calculation apparatus 701 loads all the sub-information to thebuffer (step S200). The calculation apparatus 701 then calculates thecategory attribution degree for every sub-information (step S201). Aftercalculating the attribution degree for one sub-information, thecalculation apparatus 701 determines whether or not the calculation ofthe attribution degree is completed for all the sub-information (stepS202). If the calculation of the attribution degree is not completed forall the sub-information (N of step S202), the process returns to stepS201, and the processes after step S201 are repeated. If the calculationof the attribution degree is completed for all the sub-information (Y ofstep S202), the sub-information is selected according to the magnitudeof the attribution degree (step S203). The calculation of theattribution degree and the selection of the sub-information may beperformed similar to the calculation and the selection described in thefirst exemplary embodiment.

The calculation apparatus 701 then loads the sequentially inputted maininformation to the buffer (step S204). The calculation apparatus 701then extracts (calculates) a feature from the main information with theselected sub-information as the feature extraction filter (step S205).This calculation may be also performed similar to the calculationdescribed in the first exemplary embodiment.

Effects similar to the first exemplary embodiment are obtained in thethird exemplary embodiment.

The present invention is applicable to feature extraction used inclassification by category of human data or correlation of human datanecessary in customer data collection in a convenience store etc.,security securement in an immigration control system, an entertainmentservice in a game center, a portable terminal application, and the like.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing a configuration example of a firstexemplary embodiment of the invention;

FIG. 2 is a flowchart showing an example of progress of the process bythe feature extraction apparatus in the first exemplary embodiment;

FIG. 3 is a block diagram showing a configuration example of a secondexemplary embodiment of the invention;

FIG. 4 is an explanatory view showing a configuration example of a maininformation input apparatus;

FIG. 5 is an explanatory view showing a configuration example of asub-information input apparatus;

FIG. 6 is an explanatory view showing a configuration example of asub-information selection apparatus;

FIG. 7 is an explanatory view showing a configuration example of acorrelation feature extraction apparatus;

FIG. 8 is an explanatory view showing a configuration example of alearning/identification apparatus;

FIG. 9 is a block diagram showing a configuration example of a thirdexemplary embodiment of the invention;

FIG. 10 is a flowchart showing an example of progress of the process ofa calculation apparatus; and

FIG. 11 is a flowchart showing an example of progress of the process ofthe calculation apparatus.

DESCRIPTION OF SYMBOLS

10 main information input device

20 sub-information input device

30 sub-information selection device

40 correlation feature extraction device

1-12. (canceled)
 13. A feature extraction apparatus for extracting, frommain information obtained from an individual piece of data, a feature ofthe individual piece; the feature extraction apparatus comprising: acorrelation feature extraction device for calculating a feature quantityof an individual piece from the main information of the individual piecebeing a target for extracting a feature, based on a correlation betweenthe main information of the individual piece and sub-information whichis different from the main information and a relationship between thesub-information of the individual piece and the category, by using thesub-information as a feature extraction filter.
 14. The featureextraction apparatus according to claim 13, further comprising: asub-information storage device for storing the sub-informationclassified into categories in advance; and a sub-information selectiondevice for calculating a category attribution degree of eachsub-information and selecting the sub-information whose attributiondegree is larger than a predetermined reference; wherein the correlationfeature extraction device calculates the feature quantity of theindividual piece based on the correlation between the main informationand the sub-information, by using the main information and thesub-information which is selected by the sub-information selectiondevice.
 15. The feature extraction apparatus according to claim 14,wherein the sub-information selection device performs a main componentanalysis of the sub-information, and calculates the attribution degreeusing a reconfiguration error in a category to which the sub-informationbeing a target for calculating the attribution degree belongs and anaverage of the reconfiguration error in categories other than therelevant category.
 16. The feature extraction apparatus according toclaim 13, wherein the correlation feature extraction device calculatesthe feature quantity using a difference between the main information andthe sub-information having the same dimension.
 17. The featureextraction apparatus according to claim 13, wherein the main informationis information easy to be acquired from all individual pieces, and thesub-information is information difficult to be acquired from allindividual pieces
 18. The feature extraction apparatus according toclaim 13, wherein the feature quantity is a relationship with thecategory of the individual piece being a target for extracting thefeature.
 19. The feature extraction apparatus according to claim 13,wherein the feature quantity is sub-information of a target forextracting the feature obtained from a relationship of the individualpiece being the target for extracting the feature and the category, andthe correlation.
 20. The feature extraction apparatus, wherein arelationship between the main information of the individual piece andthe category is extracted based on a correlation of the main informationof a certain individual piece and sub-information which is differentfrom the main information, and a relationship of the sub-information ofthe individual piece and the category.
 21. The feature extractionapparatus according to claim 20, wherein the main information isinformation easy to be acquired from all individual pieces, and thesub-information is information difficult to be acquired from allindividual pieces.
 22. A feature extraction apparatus for extracting,from main information obtained from an individual piece of data, afeature of the individual piece; the feature extraction apparatuscomprising: a correlation feature extraction means for calculating afeature quantity of an individual piece from the main information of theindividual piece being a target for extracting a feature, based on acorrelation between the main information of the individual piece andsub-information which is different from the main information and arelationship between the sub-information of the individual piece and thecategory, by using the sub-information as a feature extraction filter.23. A feature extraction method for extracting a feature of anindividual piece from main information obtained from an individual pieceof data; wherein a feature quantity of an individual piece is calculatedfrom the main information of an individual piece being a target forextracting a feature by a correlation feature extraction device based ona correlation between the main information of the individual piece andsub-information which is different from the main information and arelationship between the sub-information of the individual piece and thecategory and using the sub-information as a feature extraction filter.24. The feature extraction method according to claim 23, furthercomprising: calculating a category attribution degree of eachsub-information classified into categories in advance and selecting thesub-information whose attribution degree is larger than a predeterminedreference; and calculating the feature quantity of the individual piecebased on the correlation between the main information and thesub-information, by using the main information and the selectedsub-information.
 25. The feature extraction method according to claim24, further comprising: performing a main component analysis of thesub-information, and calculating the attribution degree using areconfiguration error in a category to which the sub-information being atarget for calculating the attribution degree belongs and an average ofthe reconfiguration error in categories other than the relevantcategory.
 26. The feature extraction method according to claim 23,further comprising calculating the feature quantity using a differencebetween the main information and the sub-information having the samedimension.
 27. The feature extraction method according to claim 23,wherein the main information is information easy to be acquired from allindividual pieces, and the sub-information is information difficult tobe acquired from all individual pieces.
 28. The feature extractionmethod according to claim 23, wherein the feature quantity is arelationship with the category of the individual piece being a targetfor extracting the feature.
 29. The feature extraction method accordingto claim 23, wherein the feature quantity is sub-information of a targetfor extracting the feature obtained from a relationship of theindividual piece being the target for extracting the feature and thecategory, and the correlation.
 30. The feature extraction method,wherein a relationship between the main information of the individualpiece and a category is extracted based on a correlation of the maininformation of a certain individual piece and sub-information which isdifferent from the main information, and a relationship of thesub-information of the individual piece and the category.
 31. Thefeature extraction method according to claim 30, wherein the maininformation is information easy to be acquired from all individualpieces, and the sub-information is information difficult to be acquiredfrom all individual pieces.
 32. A feature extraction program for causinga computer for extracting a feature of an individual piece from maininformation obtained from an individual of data to: calculate a featurequantity of an individual piece from the main information of theindividual piece being a target for extracting a feature based on acorrelation between the main information of the individual piece andsub-information which is different from the main information and arelationship between the sub-information of the individual piece and thecategory, by using the sub-information as a feature extraction filter.33. The feature extraction program according to claim 32, furthercausing the computer to execute: a sub-information selection process ofcalculating a category attribution degree of each sub-information andselecting the sub-information which attribution degree is larger than apredetermined reference; and calculate the feature quantity of theindividual piece based on the correlation between the main informationand the sub-information, by using the main information and the selectedsub-information in a correlation feature extraction process
 34. Thefeature extraction program according to claim 33, further causing thecomputer to: perform a main component analysis of the sub-information,and calculate the attribution degree using a reconfiguration error in acategory to which the sub-information being a target for calculating theattribution degree belongs and an average of the reconfiguration errorin categories other than the relevant category in the sub-informationfeature selection process.
 35. The feature extraction program accordingto claim 32, further causing the computer to calculate the featurequantity using a difference between the main information and thesub-information having the same dimension in the correlation featureextraction.
 36. The feature extraction program according to claim 32,wherein the main information is information easy to be acquired from allindividual pieces, and the sub-information is information difficult tobe acquired from all individual pieces.
 37. The feature extractionprogram according to claim 32, wherein the feature quantity is arelationship with the category of the individual piece being a targetfor extracting the feature.
 38. The feature extraction program accordingto claim 33, wherein the feature quantity is sub-information of a targetfor extracting the feature obtained from a relationship of theindividual piece being the target for extracting the feature and thecategory, and the correlation
 39. The feature extraction program,wherein a relationship between the main information of the individualpiece and the category is extracted based on a correlation of the maininformation of a certain individual piece and sub-information which isdifferent from the main information, and a relationship of thesub-information of the individual piece and the category.
 40. Thefeature extraction program according to claim 39, wherein the maininformation is information easy to be acquired from all individualpieces, and the sub-information is information difficult to be acquiredfrom all individual pieces.
 41. A category classification method forperforming category classification of an individual piece, the methodcomprising: detecting an element of sub-information effective incategory classification from the sub-information, which is informationof different type from main information and having greater informationamount than the main information, when classifying an individual pieceinto categories based on the main information of the individual piece;detecting an element of the main information effective in categoryclassification from a relationship of the sub-information and the maininformation and the element of the effective sub-information; andanalyzing the main information based on the element of the effectivemain information.
 42. A feature extraction selection method comprising:detecting an element of sub-information effective in categoryclassification from the sub-information, which is information ofdifferent type from main information and having greater informationamount than the main information; and detecting an element of the maininformation effective in category classification from a relationship ofthe sub-information and the main information and the element of theeffective sub-information.
 43. A feature extraction apparatus comprisinga correlation feature extraction device for extracting a featurequantity related to attribution to a category of an individual piecefrom main information obtained from a certain individual piece using afirst correlation of the main information and sub-information, which isinformation of different type from the main information and havinggreater information amount than the main information, and a secondcorrelation of the information related to the attribution to thecategory of each of a plurality of individual pieces and thesub-information obtained from the plurality of individual pieces. 44.The feature extraction apparatus according to claim 43, wherein thesub-information is configured by a plurality of elements; asub-information selection device for calculating a category attributiondegree of each sub-element or a combination of the elements usinginformation related to the category attribution degree of each of theplurality of individual pieces, and selecting an element or acombination of elements whose attribution degree is greater than apredetermined reference is arranged; and the element or the combinationof elements selected by the sub-information selection device becomes thesub-information used in the first correlation and the second correlation45. The feature extraction apparatus according to claim 44, wherein thesub-information selection device uses a main component analysis resultof the sub-information as the element
 46. The feature extractionapparatus according to claims 43, wherein the main information isinformation acquired from the plurality of individual pieces, and thesub-information is information acquired from one part of the pluralityof individual pieces.
 47. The feature extraction apparatus according toclaims 43, wherein the correlation feature extraction device calculatesa feature quantity based on a difference between the main informationand the sub-information having the same dimension.
 48. A featureextraction apparatus comprising a correlation feature extraction meansfor extracting a feature quantity related to attribution to a categoryof an individual piece from main information obtained from a certainindividual piece using a first correlation of the main information andsub-information, which is information of different type from the maininformation and having greater information amount than the maininformation, and a second correlation of the information related to theattribution to the category of each of a plurality of individual piecesand the sub-information obtained from the plurality of individualpieces.